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THE POLLS IN PERSPECTIVE 


INTRODUCTION 


Public opinion polls sometimes show abrupt changes’ in 
Support for political parties or attitudes towards a policy issue. 
Different polls may also find very different patterns of support. For 
example, Angus Reid Associates reported on 10 December 1986 that the 
Liberal Party had the support of 41% of decided voters, the Conservatives 
31%, and the NDP, 28%. They also found that the NDP had 32% voter support 
in Quebec. The following day, the Gallup poll reported the Liberals as 
having 45% of the national vote, the Conservatives 30%, and the NDP 25%; 
this poll, however reported that the NDP had only 16% support in Quebec. 
Both polls had been conducted between 3 December and 7 December 1986. The 
findings of at least one of them must be wrong. How should we interpret 
such results? 

Two separate issues are of concern in interpreting the 
results of public opinion polls: the validity of any particular poll and 
the reliability of observed trends in a series of polls.(!) Opinion 
polling organizations have considerably refined their techniques over the 
past two decades and they are not prone to the kinds of gross errors found 
in earlier years when unscientific, non-random polling techniques were 
common. Nevertheless, four inevitable sources of inaccuracy remain: 
sampling error, non-sampling error, question phrasing, and the situation 


confronting the survey respondent. 


(1) A fuller discussion is available in John C. Terry and Richard J. 
Schultz, “Canadian Electoral Behaviour: A Propositional Inventory" in 
Orest Kruhlak, et al., The Canadian Political Process, Revised 
Edition, Holt, Rinehart and Winston, Toronto, 1973, p. 248-285. 
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2 
SITUATIONAL BIAS 


A. Perceptions 


An answer to a question is not simply an expression of 
opinion - it is also a response to: several other stimuli. These include 
the respondent's mood, the interviewer's attitude, the respondent's 
reaction to the interviewer, how anxious the respondent is to conclude the 
interview (which affects how much thought he or she will give to the 
answer) and so on. A critical factor is the respondent's perception of the 
situation to which, or context in which, the question applies. 

The situational factor particularly affects polls on voting 
intentions between election campaigns. Typically, respondents are asked: 
“If a Canadian federal election were held today, which one of the following 
parties would you vote for?" The undecided are then asked: "Perhaps you 
have not made up your mind; is there nevertheless a party you might be 
inclined to support?" Those expressing an inclination are then counted 
as decided voters. The fact is, however, that an election is not being 
held that day and very few citizens are unaware of that. Voters who 
consistently vote for the same party regardless of campaign effectiveness, 
leadership changes, scandals, or the identity of candidates, are giving 
reliable answers. Those who are affected by such things may vote as they 
say they will, may change their minds, or may not vote at all when election 
day comes, In this instance, people's perceptions of a non-existent 
situation affect the accuracy of a snapshot picture of the electorate at a 


given point in time. 


B. The Changing Electorate 


The moving picture of the electorate - that is, of trends in 
party support over time - is affected by a different factor: the change 
in the electorate. We conclude too easily that change in party support in 
the country as a whole or in a particular region means that voters have 
changed their minds. Some, often many, undoubtedly have (and may well 
change them again before the next election). We forget, however, that 


LIBRARY OF PARLIAMENT 
BIBLIOTHEQUE DU PARLEMENT 


One component of change is change in the electorate itself. People die, 
hundreds of thousands of them, in a normal period of about four years from 
one election to the next, and are replaced by many hundreds of thousands of 
young people who come of voting age. If we anticipate an election two 
years hence, it might make sense to include 17 and 18-year olds in the 
sample, instead of polling only those of voting age today. 

People also move; some move from one province tu another, 
affecting regional electoral preferences; some move out of the country. At 
the same time, a number of immigrants become citizens or landed immigrants 
and thereby acquire voting rights. The net result is that changes in party 
Support, or in attitudes to issues, partly reflect some people's changing 
views but also partly reflect changes in the voting population whose 
collective views we are trying to gauge. 


QUESTION PHRASING 


The woncasing Off "aw questions wital tn” opinion spolinng. 
Good questionnaire design is a combination of science and art. The 
question is the stimulus the respondent must confront and the answers can 
only be as meaningful as the question itself. Question wording is unlikely 
to affect assessment of party support; one cannot get much clearer than 
asking which party a respondent intends to vote for. The wording of other 
questions, however, certainly affects the opinions expressed in the 
answer. Two cases in point are views about foreign ownership and concern 
about inflation and unemployment. 

In 1983, Decima Research asked in a survey whether the 
federal government should legislate to encourage greater Canadian ownership 
of industry. Eighty per cent of respondents said yes. They were then 
asked whether the government should limit foreign ownership of Canadian 
industry. Only 60% agreed. When asked whether we should limit foreign 


LIBRARY OF PARLIAMENT 
BIBLIOTHEQUE DU PARLEMENT 


4 


ownership if this would result in jobs lost, only 20% agreed.(2) What 
do these results tell us about Canadian economic nationalism? 

In its Quarterly Reports from 1980 to 1983, Decima reported 
results on the question "In your opinion, what is the most important 
problem facing Canada today - in other words, the one that concerns you 
personally the most?" As one might expect, inflation was the primary 
concern in pre-recession times, selected by as many as 36% of respondents 
in the second quarter of 1981. As the recession grew and unemployment 
rose, concern about unemployment rose dramatically; inflation was selected 
as primary concern by only 9% in the last quarter of 1983. At the same 
time, Goldfarb was also examining concern about economic issues. Instead 
of asking, like Decima, respondents to choose one issue of concern, 
Goldfarb asked them to rate several issues on an index ranging from 0 to 
100, with 100 representing the highest level of concern.(3) Like Decima, 
Goldfarb found unemployment the object of most concern in 1983, with an 
average ranking of 95 points - but inflation was a close second with an 
average ranking of 89, : 

As one expert on Canadian public opinion concluded: 

Our data ... dramatize a problem which pervades survey 
evidence on public opinion: it is folly to act on only 
one reading of opinion in a domain, especially if that 
opinion, whether it be general or specific, is evoked 
in a relatively context-free way ... [Elven individuals 
with real, stable and consistent underlying attitudes 


will respond somewhat randomly to _ poorly-worded 
items.(4) 


(2) Allan Gregg, "Politics by Poll", transcript of interview on Realities, 
TV Ontario, 14 November 1985, p. 7. 


(3) Goldfarb Report, 1984, p. 39. 
(4) Richard Johnston, Public Opinion and Public Policy in Canada, Vol. 35, 


Background Studies of the Royal Commission on the Economic Union and 
Development Prospects for Canada, University -of Toronto Press, 
NOGONLO@ELOGO SED emc Lc ema Lon ‘ 
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NON-SAMPLING ERROR 


Other forms of error, such as those in recording, coding, 
and processing responses, do not result from the sampling process itself. 
Certainly every effort is made to detect, avoid, and correct technical and 
mechanical mistakes, but equally certainly they still occur. An 
entertaining example was seen in a sample survey conducted in a Vancouver 
constituency in the mid-1960s. The same respondents were interviewed 
twice, once in 1963 and again in 1965; analysis revealed that, according to 
the coded survey data, 1.8% of them had changed gender between the time of 
the first interview and the second.(5) A specialist in survey design 
cautions that “it behooves the research worker to remain severely critical, 
to search out biases in others and in himself, and to avoid giving the 
appearance of spurious exactitude." (6) 


SAMPLING ERROR 


Two kinds of survey error result from the sampling method 
itself. Most people are already familiar with the first, though they 
consistently ignore it: this is the fact that, as the standard disclaimer 
in published poll results points out, “a survey with a sample of this size 
is accurate within plus or minus 4%, 19 times out of 20". Ents. is 
statistical or mathematical error derived from probability theory. The 
second error is caused by variations in the sampling method or the sampling 
"frame", as explained in Section C below. 

Statistically, the margin of error in a survey depends on 
the size of the sample (regardless of the size of the population, Unless 
that is very small) and the "homogeneity" of the sample; that is, the 
closer the population approaches unanimity on an issue, the lower the 


(5) J.A. Laponce, eats vs. Politics, University of Toronto Press, 
TOGONLO.s LoOI sm : 


(6) A.N. Oppenheim, Questionnaire Design and Attitude Measurement, Basic 
BOOKS ENE WRTOLK me LGOOmmD er cL. 
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margin of error. On some issues, homogeneity may significantly reduce 
error - the vast majority of the population (and of any sample), for 
example, are opposed to war and in favour of peace. When it comes to party 
Support, however, the public displays very little homogeneity and for 
practical purposes the margin of error is determined purely by sample 
size. Decreasing the margin of error to a dramatically low level requires 
extremely large samples at a very high cost. To reduce error to less than 
+ 1%, for instance, would require more than 22,500 interviews. We do get 
occasional surveys of 1,500 to 2,000 respondents with reported error 
margins of between + 2.7% and + 3.2%, but most surveys claim an accuracy of 


only + 4%, with a sample of about 1,000 people. 


A. National Trends 


Figure 1 shows trends in party support since the 1984 
federal election, as shown in the monthly Gallup reports. These reports 
over the past year have been accompanied by newspaper headlines proclaiming 
"Liberals take the lead", "Conservatives regain the lead", "NDP surges in 
popularity", “Liberal lead widens", and so on. How much truth there is to 
these claims is indicated in Figures 2, 3 and 4, Figure 1 illustrates 
support trends with a shaded confidence interval of + 4%. Here we may see 
clearly that there was an emphatic decline in Conservative popularity 
during 1985 and a rise in support for the Liberal Party. However , 
statistically, at no point until December 1986 could the Liberals be 
confidently said to have taken a lead. Throughout 1986, until the December 
polls, the differences between the two leading parties were within the 
bounds of the margin of error. There were three occasions in the latter 
half of 1986 when Conservative support dipped below the shaded confidence 
interval shown. If we can imagine the confidence interval as a curve, 
however, rather than a straight line, with its upper edge at the level of 
the "leading" party, then the level of Conservative support would have been 
inside the confidence interval until the most recent polls. At one point, 
in July 1986, the data could have been interpreted as indicating that any 
of the three parties was in the lead. 
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Figure 4 
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Figure 2, showing Conservative support in the Gallup polls, 
emphasizes the drop in the government party's support during 1985 but 
offers little reinforcement for any argument that it continued to drop in 
1986, until the two most recent polls. Similarly, the pattern of Liberal 
support shown in Figure 3 offers no evidence of increased Liberal support 
from the fall of 1985 until the surge recorded in December 1986, a surge 
that was arguably affected by the events and publicity surrounding the 
Liberal convention. In the most recent poll, January 1987, the Liberal vote 
is back within the bounds of the confidence interval. Finally, the data 
in Figure 4 offer no statistical evidence whatsoever of increased - or 
decreased - support for the NDP over the past two years. The one-poll 
surge for that party in the summer of 1986 gives the appearance of being 
precisely what one would statistically expect: the one poll out of twenty 
that is wrong. 


B. Regional Trends 


When national poll results are published we are inundated 
with regional breakdowns of party support as well - and so we should be, 
since, because of regional concentrations of support, national support 
for a party does not necessarily imply that it will win seats. As we are 
not told the sample sizes for each region, however, we are left with the 
impressions thats the srepontedmeacciumacymmleve| of +542 applies @to calt 
percentages quoted in the analyses. That is at best an accidental omission 
and at worst a deliberate attempt to mislead. Failure to report regional 
Samplew si zesmeands their, marginsSs Otmmercnor smay saccount | TOrmethe recent 
journalistic controversy over NDP support in Quebec. In December 1986, 
Gallup showed Quebec NDP support at 16%, (down from 33% in October), Angus 
Reid showed the NDP at 32% (up from 26% in November), and Environics 
reported NDP support to be 25% (down 1% since October). In January 1987, 
Gallup found 30% NDP support in Quebec, and Reid reported 29% support. 

The 1984 Goldfarb Report gives a breakdown of regional 
sample sizes as follows for a total national unweighted sample size of 
1,497~ |S Britisnacolumbia, l5ieeePprainiess. 2663. Ontario, 516;sQuebec,. 398 
(60 of them English-speaking); and the Maritimes, 166. A sample of 
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approximately 150 (British Columbia and the Maritimes) has a margin of 
error of + 8.2%; the Prairie sample of 266 has a margin of error of + 6.2% 
(and a much higher margin of error for statements about individual Prairie 
provinces, which have less than 100 respondents each); the Quebec margin of 
error would be over + 5.0% (and it is ludicrous to infer the views of 
English-speaking Quebeckers from a sample of only 60); the margin of error 
for Ontario would be + 4.5%. In a smaller sample, such as Gallup's, the 
regional samples would be correspondingly smaller and the margins of error 
larger. In light of this, one talks of changes or trends in party support 
in a single province at one's peril. Given the margins of error for Quebec 
in the different samples, the answer to the question posed in the 
introduction - which poll is wrong and which right about Quebec - must be: 
all of them or none of them, Figure 5 displays recent reports of NDP 
Support in Quebec with a + 6% confidence interval. 


C. Sampling Frames 


Finally, a word must be said about sampling frames, that is, 
the structure of the national survey samples. The margins of error 
discussed above are, mathematically, based on a pure random sample of the 
population. Because of the dispersal of Canada's population in remote 
areas, however, a pure random sample would be prohibitively expensive. 
Accordingly, adjustments are made. 

Gallup does not include a statement of sampling methods in 
its reports, but Goldfarb and Decima do. Goldfarb conducts in-home 
interviews, drawing its sample from a population that excludes the Yukon, 
the Northwest Territories, Indian Reserves, and all remote or sparsely 
populated areas. Some inevitable biases are built into the sampling 
process, apart from the exclusion of regions. In 1984, one-quarter of the 
people Goldfarb tried to contact could not be reached at home; of those who 
were contacted, one-half refused to participate in the survey. People who 
are rarely at home or who refuse to answer surveys may have characteristics 
which differentiate them from those who are at home and willing to 
participate. This fact may contribute to distorted poll results. 
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Decima conducts its interviews by telephone, sacrificing 
personal contact for wider geographical contact - its sample excludes the 
Yukon and the Northwest Territories but includes all the remaining Canadian 
population with telephones. However, in Saskatchewan, Quebec, and Ontario 
outside of Toronto, Decima's telephone sampling is based on telephone 
directories, so that those with unlisted or newly acquired telephone 
numbers are excluded. Telephone interviewing also excludes people who are 
rarely at home, thereby, like at-home interviews, perhaps missing people 
with certain characteristics. 

Al) Canadian public opinion surveys intentionally 
“oversample' the more sparsely populated parts of the country in order to 
have at least a reasonable number of interviews in each province. This 
means that the reported survey results for the nation are based on weighted 
samples, that is, the figures are adjusted to reflect a sample as if it 
were randomly drawn from the population. For example, in 1984 the 60 
people in Prince Edward Island whom Decima interviewed, constituted 4.0% of 
the total sample, although Prince Edward Island has only 0.5% of the 
Canadian population. A random sample of the total size used by Decima 
(1,500) would have included only eight residents of Prince Edward Island. 
In calculating national averages, Prince Edward Island respondents were 
"weighted" at 0.1333 out of 1.0. Ontario outside of Toronto constituted 
26./% of the population in the 1981 census, but only 14% of Decima's 
interviews came from that area - so in national figures, their views were 
weighted as 1.90 out of 1.0. That sort of weighting procedure does not 
drastically affect national estimates, but may seriously affect regional 
projections. Survey reports do not state whether regional estimates are 
based on weighted or unweighted samples. If they are based on weighted 
samples, then we are dealing with incredibly small sample sizes, with very 
large margins of error. If they are based on unweighted sample sizes, it 
is meaningless to compare the unweighted figures for regions or provinces 
with weighted national figures which take into account those regions or 


ra 


provinces. 
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CONCLUSION 


Despite the problems inherent in survey methodology, public 
opinion polls have a commendable record of accuracy, at least as measured 
against election returns. Polls conducted just prior to elections by 
reputable firms are rarely (perhaps once in 20 times) far off the mark. 
There is no reason to expect that they should be any more erroneous in 
their estimates of opinions or voters' intentions between elections. In 
interpreting poll findings, however, we should be aware of sources of 
potential error in them. Particularly with regard to election 
prognostications, we should be mindful that the polls are assessing a 
merely hypothetical situation. For those who keep these limitations in 


mind, public opinion surveys can be useful guides to action. 
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